Multivariate Models

IS 5150/6110

Project Update

  • Add me and Holden as collaborators on each of your project repositories
  • Create the PR for that week’s branch to submit the associated milestone
  • Once possible changes have been made to the milestone, we will merge
  • Access to data should be a serious focus at this point in the semester
  • However, you could use experimental or observational data

Probabilistic Perspective

  • Using probability to quantify uncertainty about what we know
  • All unknowns are random variables (i.e., can have probability distributions)
  • We have joint distributions \(p(x, y)\) and marginal distributions \(p(x)\)
  • Lots of moments or summary statistics, but they have limited information
  • Bayes’ rule provides a principled way to update what we know

\[ \LARGE{p(\theta | X) \propto p(X | \theta) \ p(\theta)} \]

\[ \LARGE{\text{posterior} \propto \text{likelihood} \times \text{prior}} \]

Frequentist

\[ \theta = \text{some value} \\ X \sim \text{some distribution} \]

Bayesian

\[ \theta \sim \text{some distribution} \\ X = \text{some value} \]

Frequentist

\[ \theta = \text{some } \color{red}{\text{unknown}} \text{ value} \\ X \sim \text{some distribution} \]

Bayesian

\[ \theta \sim \text{some distribution} \\ X = \text{some } \color{red}{\text{known}} \text{ value} \]

Multivariate Models

What questions do you have?

  • \(x\) scalars
  • \(\mathbf{x}\) vectors (i.e., column vectors)
  • \(\mathbf{X}\) matrices, composed of rows and columns
  • Bonus: Python apparently fills matrices by rows

Covariance is a multivariate generalization of variance. Correlation is a normalized covariance (i.e., scaled to \([-1, 1]\)).

\[ \text{Cov}[\mathbf{x}] = \mathbf{\Sigma} = \begin{bmatrix} \text{Var}[X_1] & \text{Cov}[X_1, X_2] & \text{Cov}[X_1, X_3] \\ \text{Cov}[X_2, X_1] & \text{Var}[X_2] & \text{Cov}[X_2, X_3] \\ \text{Cov}[X_3, X_1] & \text{Cov}[X_3, X_2] & \text{Var}[X_3] \end{bmatrix} \]

Multivariate models are generalization of their univariate counterparts. Why do we need the added complexity? What is a generalized linear model?

  • Multivariate Gaussian/Normal (including Bivariate)
  • Exponential family (maximum entropy models)
  • Mixture models (oh my!)

We can express any probabilistic model (i.e., joint distribution) as a graph where conditional independence and causal structure is encoded.

  • Empty nodes are observed
  • Shaded nodes are unobserved
  • Edges are the lines connecting nodes
  • Arrows show the flow of information (causality)
  • We are especially interested in directed acyclic graphs

::::

“It’s only confusing because you’re paying attention.” -Richard McElreath